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TWO EXPERIMENTS WERE CONDUCTED TO EVALUATE A SELF-STUDY 
PROGRAM PREPARED WITHIN TWO OPERATIONAL FIGHTER-INTERCEPTOR 
SQUADRONS. BASED ON A TECHNICAL ORDER MANUAL, IT CONSISTED OF 
SEQUENCED MULTIPLE-CHOICE QUESTIONS WITH A SIMPLE FUNCHBOARD 
AS A CONFIRMING DEVICE. IN EXPERIMENT ONE, TWO OPERATIONAL 
SQUADRONS WERE DIVIDED INTO TWO GROUPS JUDGED TO BE AT 
COMPARABLE LEVELS OF KNOWLEDGE AS BASED ON A PRETEST. DURING 
THE 11 DAYS OF THE EXPERIMENT, THE EXPERIMENTAL GROUP SPENT 
JUST OVER FOUR HOURS ON SELF-INSTRUCTION, MOSTLY GOING OVER 
THE PROGRAM TWICE, WHILE THE CONTROL FOUND ITS CLASSROOM 
LECTURES INTERRUPTED BY TEMPORARY DUTY AND EXTENDED ALERTS 
AND SPENT SLIGHTLY MORE THAN ONE HOUR. THE EXPERIMENTAL GROUP 
DIB BETTER ON THE POST-TEST AND LIKED THE PROGRAM AND IT WAS 
COMPLETELY WITHIN THE CAPABILITIES OF THE SQUADRON TO PREPARE 
THE MATERIALS. IN THE SECOND EXPERIMENT, THE TIME WAS 
CONTROLLED. THE EXPERIMENTAL GROUP WENT OVER THE PROGRAM 
TWICE, SPENDING AN AVERAGE OF 253 MINUTES WHILE THE CONTROL 
GROUP WAS GIVEN 250 MINUTES OF LECTURE OVER FIVE BAYS. AGAIN 
The EXPERIMENTAL GROUP SCORED BETTER ON A POST- TEST, OVER 
ONE-FOURTH SCORING AS HIGH OR HIGHER THAN THE HIGHEST IN THE 
CONTROL GROUP. (EB) 
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AN adjunct; to self-study 

DONALD E. MEYER 

Behavioral Sciences Laboratory 
Wright'Patterson AFB, Ohio 




T he task of an aircrew member in 
the modern defense structure is a 
highly complex one. Training toward 
mastery of the required skills and related 
knowledges is a long, arduous, and costly 
process. 

A portion of the learned repertoire 
falls into a unique category. It consists 
of bits of information and behaviors that 
are not frequently used or may be in the 
nature of isolated factual information 
that does not lend itself to structuring 
for simple recall. Certain engineering 
features, some flying regulations, alterna- 
tive procedures for weapons deployment 
due to malfunction, etc., fall in this cate- 
gory. While much of this seldom-needed 
part of the repertoire is not of a critical 
nature, some of it is. Circumstances may 
occasionally combine in such a way that 
their fluent use could mean the difference 



Doctor Meyer (bachelor’s and master's degrees, 
Colorado State; EdD, Arizona State), loined the 
Behavioral Sciences Laboratory, Training Re- 
search Division, in 1963, and is presently assigned 
to the Operator Training Branch. He has been 
closely associated with Air Force flying training 
since his experiences as a pilot during World War 
II. Doctor Meyer has authored programmed ma- 
terials for both ATC and TAC. 



between disaster or safety, or an abort or 
successful mission. 

Retr )shar Training 

Many of the skills in which an aircrew 
member is trained are practiced almost 
daily in normal mission performance. In 
these practiced areas, skill level and con- 
cept retention remain high. In other 
areas, where a task is performed only 
intermittently, and even then at various 
frequencies by the different aircrews or 
aircrew members, a loss in proficiency is 
an inevitable result. It is consequently 
necessary to conduct refresher training at 
prescribed intervals and on a continuing 
basis to maintain proficiency at a desired 
level. 

Typically, conventional classroom 
methods are used in presenting refresher 
training. Lectures by training officers, 
factory technical representatives, and oc- 
casional guest appearances by recognized 
experts in a given field are the approaches 
most frequently used. 

In light of modern educational tech- 
nology, however, each of these ap- 
proaches has deficiencies. Instructor lim- 
itations, the lack of any great amount 
of student participation, a training sched- 
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ule dictated by operational mission re* 
quirements, and difficulties in obtaining 
! training aids and graphics are some gen- 

erally recognized factors limiting the 
; effectiveness of refresher training in op- 

erational units. 

i Improvement Needed 

This nuclear age with its constant 
threat of vast destruction has established 
, a requirement for the dispersal of opera- 

tional units to insure a continuing defense 
and retaliatory ca acity should an attack 
materialize. In view of this fact, the 
' trend toward widespread dispersal is a 

growing reality, and as the trend con- 
tinues, the magnitude and difficulty of 
conducting refresher training by conven- 
, tional means becomes more and more 

impractical. 

Programmed instructional techniques 
have been suggested as ways to provide 
refresher training in operational squad- 
rons. However, an analysis of the more 
familiar techniques reveals some serious 
deficiencies for this application. The pur- 
pose of refresher training is review of the 
subject material to clarify and emphasize 
both the critical and not so critical in- 
formation in an attempt to assure accept- 
able skill and knowledge levels. The air- 
crew members who comprise the student 
population are already familiar with the 
subject matter areas. 

* However, a typical cross section of , 

. aircrews reveals considerable variance in 

background and experience levels, as well 
■ as recency of formal training, personal 

' habits and capabilities. It is evident that 

I proficiency is scattered across a consider- 

j. able range. The use of one or more of 
the familiar programming approaches ap- 
1 pears inappropriate in that when detailed 

! enough for the lower proficiency levels, 

it is much too basic for the more profi- 
1 cient students unless extensive branching 

is used. Also, the more familiar program- 






ming techniques require traii^ed program- | 

mers and extensive time to prepare. 

Self-Study Program 

An adaptation of self-testing teaching | 

technique appeared promising, and after | 

considerable consultation with expert j 

crew members the technique was selected | 

for experimental trial. The technique is | 

illustrated by FIGURE 1 . | 

The self-study program consists of a j 

comprehensive and carefully sequenced j 

series of multiple-choice questions de- |. 

signed to cover all of the important in- 
formation contained in a technical order 
(TO) manual. Listed with each question 
is the exact page and paragraph number 
in the TO from which the question was 
derived. 

A sintple punchboard is used as the 
confirming device. If the student knows 
the subject information and responds to 
the right answer it is confirmed as correct i 

by the punchboard and the student goes I 

on to the next question. If the student | 

responds incorrectly on the punchboard \ 

he is directed to the page and paragraph | 

number in the TO manual for the “reme- \ 

dial” instruction needed to respond cor- ; 

rectly, after which he goes on to the r 

next question. ; 

Number of Advonfoges \ 

• 

A number of advantages appear in- \ 

herent in this technique that seem par- | 

ticularly suitable to the existing training ^ 

situation; { 

• The self-instructional sequences; i.c., 

the multiple-choice questions, can be 
quickly made up at squadron level by op- 
erational personnel without extensive 
training or practice in programming 
techniques. > 

• The punchboards and scoring keys j 

can be inexpensively manufactured by | 

squadron personnel, using re£idily avail- 
able base facilities. 
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Fi 9 ur* 1 Diagram of Adjunct Mathod 



• It offers the student a means to 
progress through the instructional se- 
quence at his own pace and in the mini- 
mum time consistent with his individual 
level of knowledge. 

• A student is “instructed" only in 
areas of deficiency, thus saving both time 
and frustration. 

• The program, punchboard, and TO 
can be conveniently carried to a place 
and used at a time of the student’s 
choosing. 

• The punchboard response-confirm- 
ing device appears to have a motivational 
effect upon those students who use it. 

Purpof* of Experimont 

Considering the potential value of the 
technique, experimental studies were con- 
ducted to evaluate the technique for 
operational use. More specifically, the 
experiments were conducted 



o to determine if the punchboard tech- 
nique of preparing self-instructional ma- 
terials is within the capability of an op- 
erational squadron; 

• to compare the effectiveness of the 
self-study technique with conventional 
classroom instruction; and 

o to obtain an estimate of student re- 
action toward the self-study approach. 

Method of Moteriolf Development 

In an attempt to assure comprehensive 
coverage of the subject matter to be pre- 
sented, the first step was to carefully go 
through a specific TO manual (TO IF- 
lOlB-29, Weapons Manual) to estimate 
by paragraph, the number of questions 
needed to thoroughly cover the subject 
matter content. Two fighter-interceptor 
squadrons were then furnished the break- 
down by page, paragraph, and number 
of questions required, together with a 
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request that each of their qualified offi- 
cers be assigned the responsibility for 
making up 10 multiple-choice, four alter- 
native-response questions covering the 
specified paragraphs and marking the 
correct answer as well as the page and 
paragraph nun^ber to which each ques- 
tion had reference. Guidelines for the 
preparation of questions were drawn up 
and furnished each officer. An extensive 
item pool of questions, suitably refer- 
enced, was thus obtained. Questions were 
selected from this pool, edited as neces- 
sary, and sequenced to form the self- 
study program. 

The item pool also provided the source 
from which 50 questions were drawn to 



make up the criterion examination. Due 
care was taken to assure that a represen- 
tative sampling of the subject matter was 
covered and also to avoid duplication of 
any item used in the instructional pro- 
gram. The same items used for the pre- 
training test were presented in reverse 
order as the post-training test. 

The punchboards used as response-con- 
firming devices were designed and fabri- 
cated by an operational squadron. The 
punchboards were designed to use metal 
slide-in answer keys to accommodate any 
number of instructional test items by 
merely changing answer keys. A concep- 
tion of the construction details can be 
obtained from FIGURE 2. 
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IXPERIMENT I 

The experiment was initiated with the 
administration of a 50-item multiple- 
choice pretraining test to the two par- 
ticipating squadrons. (Henceforth, the 
squadron given instruction by the self- 
study method will be referred to as the 
experimental (E) group; the squadron 
receiving conventional classroom instruc- 
tion will be referred to as the control 
(C) group.) The purpose of the pretest 
was to estimate the comparability of the 
two groups on the basis of knowledge. 
The results are summarized in TABLE 1. 



Table 1 

Experiment I pretest 





E 


C 




Group 


Group 


Number of subjects 


34 


30 


Mean raw score 


34.41 


32.73 


Mean score in percent 


68.82 


65.46 


Standard deviation 


4.76 


5.84 


Standard error (mean) 


.82 


1.07 



A t-test of statistical significance was 
applied to these data. The obtained t- 
value indicated no significant differ- 
ence between the means of the two 
groups.* Upon this basis the Iwo groups 
were judged to be at comparable levels 
of knowledge prior to the training 
prescribed. 

Instruction Time 

Subjects in the experimental group 
were instructed to work thiough the self- 

* EDITOR'S NOTE: A »-test is a statistical procadure 
used to determine whether the difference between the 
means of two groups is real and not an observation 
obtained by chance. With the use of prepared tables, 
the t-value is evaluated in terms of the probability of 
its occurring by chance atone. Since the t-value reflects 
the degree of difference between the two means, the 
significance level of .001 means that only once in 1000 
would such o difference be expected to occur by chance 
alone. The difference, therefore, is attributed ta the 
difference between the experimental and control 
groups. 



study booklet at least once during the 
1 1, days scheduled for thq experiment. 
Personnel in the control gro'up were to 
be scheduled for classroom instruction 
covering all of the subject material dur- 
ing this period. 

Members of both groups were in- 
structed to keep accurate records of the 
amount of time individually spent in 
their respective modes of instruction. Al- 
though the resulting records are probably 
more in the category of time estimates 
than precise measures, they are included 
in this report to form a basis for com- 
parison. Subjects in the experimental 
group averaged four hours and 13 min- 
utes of self-study compared to an average 
of one hour and 1 3 minutes of classroom 
instruction for subjects in the control 
group. Most of the subjects in the experi- 
mental group reported going through their 
self-study materials twice. 

A number of factors intervened which 
interrupted classroom schedules for the 
control group (temporary duty, extended 
alerts, assignments to ferry aircraft, etc.). 
This, however, is a normal condition in 
an operational squadron faced with mis- 
sion responsibilities. The experimental 
group, though faced with the same re- 
sponsibilities, was able to devote an aver- 
age of almost four times more time on 
instruction than the control group due 
to the self-study feature: i.e., subjects 
could study at a place and time of their 
own individual choice, independent of 
any group effort or fixed schedule. 

Post-training Test 

A post-training test was administered 
to both groups 11 days after initiation of 
the project. The results are summarized 
in TABLE 2. 

The 3.95 t-value obtained from these 
data indicates a difference between means 
that is significant beyond the .001 in 
favor of the experimental group. 
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Immediately following the post-train- 
ing test, subjects in the experimental 
group were asked to complete a ques- 
tionnaire in an attempt to gain an esti- 
mate of the degree of acceptance of the 
self-study technique. Most of the re- 
sponses indicated a high degree of ac- 
ceptance and enthusiasm for the self- 
instructional technique'. 

Table 2 



Experiment I post-training test 





E 


C 




Group 


Group 


Number of subjects 


34 


30 


Mean score 


41.44 


35.97 


Mean score in percent 


82.88 


71.94 


Standard deviation 


4.55 


6.27 


Standard error (mean) 


.784 


1.143 



Experiment I Findings 

The findings of this experiment indicate 

• the preparation of the punchboards 
and self-instructional materials is well 
within the capabilities of an operational 
squadron; 

• the self-study technique is superior 
to conventional classroom refresher train- 
ing as measured by the post-training ex- 
amination; and 

• the subjects who participated in the 
post-training program were generally en- 
thusiastic in support of the self-study 
method. 

The disproportionate study times (one 
hour and 13 minutes conventional, four 
hours and 13 minutes self-study) are 
realistic comparisons that emphasize the 
difficulty of conducting group instruction 
under operational conditions. The fact 
that almost four times more study time 
was available to subjects engaged in self- 



study is a decided advantage over con- 
ventional instruction. From an opera- 
tional viewpoint the study time disparity 
marks the self-study method as definitely 
advantageous; experimentally, however, 
a question arises concerning how well the 
self-study method would compare if the 
same amount of time were devoted to 
classroom instruction. On this basis a 
decision was made to repeat the experi- 
ment holding the time factor constant. 

EXPERIMENT II 

Two operational squadrons were again 
identified as experimental participants. 
As in the first experiment, the squadron 
assigned the self-study technique will 
be referred to as the experimental group, 
with the conventionally taught squadron 
as the control group. Both groups partic- 
ipating in the experiment were pretested 
using the same 50-iiem multiple-choice 
test used in the first experiment to deter- 
mine the comparability of knowledge 
levels prior to training. The results of 
the pretraining test are summarized by 
TABLE 3. The two groups were judged 
comparable upon finding a nonsignifi- 
cant t-value. 

The control group was then given a 
50-minute instructional period each day 
for five consecutive days. The lecture 
method was utilized for this instruction 
by a well-qualified fighter-interceptor 
weapons instructor. 

Table 3 



Experiment II pretraining test 





E 

Group 


C 

Group 


Number of subjects 


26 


24 


Mean score 


34.19 


32.04 


Mean score percent 


68.38 


64.08 


Standard deviation 


4.89 


4.62 


Standard error (mean) 


.96 


.94 
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Camparabl* Sturfy Timai 

Subjects in the experimental group 
were directed to work through the self- 
study program twice during the experi- 
mental period and to individually maintain 
accurate records of the amount of time 
so spent. It had been determined in the 
previous experiment that approximately 
250 minutes were required. The result 
was comparable training times for the 
two groups. Subjects in the experimental 
group spent an average of 253 minutes 
in self-study, compared to 250 minutes 
of classroom instruction for the control 
group. 

The post-training test was administered 
to both groups at the completion of the 
instruction periods. The results are 
shown in TABLE 4. 



Tabu 4 

Exparlmtnt II peiMralnIng t«it 





E 


C 




Group 


Group 


Numbtr of lubjecti 


26 


24 


Mtan score 


40.89 


35.79 


Mean score percent 


81.78 


71.58 


Standard deviation 


3.79 


5.24 


Standard error (mean) 


.74 


1.07 



An examination of the frequency dis- 
tributions of the scores obtained by the 
two groups reveals some interesting 
differences. Twenty-seven percent of the 
subjects in the experimental group scored 
as high or higher than the highest score 
made in the control group. Only 17 
percent of the control group scored at 
or above the mean of the experimental 
group while 88 percent of the experi- 
mental group scored above the mean of 
the control group. Twenty-one percent 
of the subjects in the control group 
scored lower than the lowest score made 
in the experimental group. The scores 
ranged from 32-49 for the experimental 
^oup compared to a range of 23-44 
for the control group. 



The obtained t-value of 3.95 indicates 
a difference between the means that is 
significant at the .001 level in favor of the 
experimental group. The gain in achieve- 
ment over the pretest was 13.40 per- 
centage points for the experimental 
group, compared to a 7.50 percentage 
point gain for the control group. 

As in the first experiment, a student re- 
action questionnaire was administered to 
subjects in the experimental group. The 
results again were generally quite favor- 
able toward the self-study technique. 



Plndingi Dlicuiiad 

The statement that **no single training 
technique or method is universally 
adapted to all training situations" would 
probably be argued by no one. There 
are, however, classes or categories of 
training situations to which certain ap- 
proaches seem particularly appropriate. 

The self-study technique used during 
this study appears suitable to training 
needs in the categories of review and 
refresher training. The use of punch- 
boards or other response confirming de- 
vices is not unique in the technology of 
training. A history of such contrivances 
is readily found in existing literature ( 1 ) . 
This study emphasizes the potential of 
this method in a “do-it-yourself" environ- 
ment. 

The modern defense structure with its 
isolated site and dispersal requirements 
is such that conventional group instruc- 
tion, after an individual is assigned to a 
command, is not practical in many cases. 
Inappropriate facilities, the nature of the 
assigned mission, limited operating bud- 
gets, and the lack of trained instructor 
personnel conspire to rule out on-site 
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operational commands appears to be of 
this type. Here the student has already 
been trained or is experienced in the 
required tasks. To be maximally effec- 
tive, instruction for this person should 
center only in areas which have been 
forgotten, partially forgotten, or confused 
through lack of use. To spend time, 
money, and effort in instructing a person 
concerning material he already knows 
is pointless and as frustrating to the stu- 
dent engaged in '>elf-study as it is to a 
student in a conventional classroom. 

Ob|v Jiens Aveidad 

The adjunct techniques used in this 
study seem to avoid many of these ob- 
jections. By utilizing existing publications 
as information sources the most expen- 
sive single cost is avoided. The questions 
and response-confirming devices can be 
quickly and easily made up without 
extensive training in programming tech- 
niques. It is suitable for students at 




Flying offieeri find the adjunct program ta be a time saving method of study. 



group instruction as an effective ap- 
proach to continuation or refresher train- 
ing. Self-instruction appears to be the 
best if not the only means of providing 
this necessary function. 

Limitod Effectivoness 

Conventional manuals, texts, and tech- 
nical orders are provided for most of the 
required fields of knowledge, but the 
mere reading, even of the best of these 
when considered as the only means of 
instruction, is of limited effectiveness. 
The new and rapidly developing field of 
programmed instruction is proving an ef- 
fective means of solving the problems 
of self-instruction. However, the task of 
programming is a costly and time-consum- 
ing process requiring considerable train- 
ing for mastery. 

Moreover, this degree of pedagogical 
thoroughness and refinement may not be 
necessary in some areas. Most refresher 
training of the kind required by the 
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heterogenous levels of knowledge. Since 
students are instructed only in areas of 
deficiency as confirmed by their responses 
and, because it provides direct reference 
to the page and paragraph number in 
the test to correct the deficiency, it is 
economical in its requirements on student 
time and effort. 

It is easily portable and can be studied 
at a time and place of the students’ own 
choosing. It is self-paced. It can be ac- 
complished in short settings since the 
steps (questions) are separate items 
rather than continuous discourse. The use 
of the punchboard seems to have motivat- 
ing aspects, apparently adding some fea- 
tures of game playing. Most students in- 
dicated they liked it as a self-study 
method. Last, and perhaps most im- 
portant, it is an effective training 
technique. 

These features provide a powerful 
argument for careful consideration of 
this technique by operational commands 
faced with refresher training require- 



ments. The cost and ease of implemen- 
tation are minimal in. view of the findings 
and conclusions of this study.' 

Conclusions Warranted 

The findings of the two experiments 
conducted during this study would seem 
to warrant the following conclusions: 

• The preparation and administration 
of the self-study technique are entirely 
Within the capability of an operational 
squadron with only a minimum amount 
of guidance. 

• In the operational setting, the self- 
study technique is superior to conven- 
tional classroom methods in its effective- 
ness as a means of presenting refresher 
training. 

• Students are favorable in their ac- 
ceptance of the self-study presentation 
of refresher training. 
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AF JUNIOR ROTC TO EXPAND 

Air Force Junior Reserve Officer Training Corps programs will be offered In 100 
high schools by fiscal year 1969. Retired Air Force officers and noncommissioned 
officers will conduct the programs. Persons expecting to retire In fiscal year 1968 and 
interested in the Junior AFROTC program should write: Commandant, Air Force ROTC 
(ARTO-J), Maxwell AFB, Ala. 36112. 
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